A Conspiracy of Random X and Model Violation against Classical Inference in Linear Regression

Following the econometric literature on model misspecification, we examine statistical inference for linear regression coefficients βj when the predictors are random and the linear model assumptions of first and/or second order are violated: E[Y |X1, ..., Xp] is not linear in the predictors and/or V [Y |X1, ..., Xp] is not constant. Such inference is meaningful if the linear model is seen as a useful approximation rather than part of a generative truth. A difficulty well-known in econometrics is that the required standard errors under random predictors and model violations can be very different from the conventional standard errors that are valid when the linear model is correct. The difference stems from a synergistic effect between model violations and randomness of the predictors. We show that asymptotically the ratios between correct and conventional standard errors can range between infinity and zero. Also, these ratios vary between predictors within the same multiple regression. This difficulty has consequences for statistics: It entails the breakdown of the classical ancillarity argument for predictors. If the assumptions of a generative regression model are violated, then the ancillarity argument for the predictors no longer holds, treating predictors as fixed is no longer valid, and standard inferences may lose their significance and confidence guarantees. The standard econometric solution for consistent inference under misspecification and random predictors is based on the “sandwich estimator” of the covariance matrix of β̂. A plausible alternative is the paired bootstrap which resamples predictors and response jointly. Discrepancies between conventional and bootstrap standard errors can be used as diagnostics for predictor-specific model violations, in analogy to econometric misspecification tests. The good news is that when model violations are sufficiently strong to invalidate conventional linear inference, their nature tends to be visible in graphical diagnostics.